On Some Extensions of the Natural Gradient Algorithm

Recently several novel gradient descent approaches like natural or relative gradient methods have been proposed to derive rigorously various powerful ICA algorithms. In this paper we propose some extensions of Amari’s Natural Gradient and Atick-Redlich formulas. They allow us to derive rigorously some already known algorithms, like for example, robust ICA algorithm and local algorithm for blind decorrelation. Furthermore, we hope they enable us to generate the family of new algorithms with improved convergence speed or performance for various applications. We present conditions for which the proposed general gradient descent dynamical systems are stable. We show that the nonholonomic orthogonal algorithm can not be derived from minimization of any cost function. We propose a stabilized nonholonomic algorithm, which preserves the norm of the demixing matrix.